Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Backend entrypoint #3166

Closed
wants to merge 8 commits into from
Closed

[Feature] Backend entrypoint #3166

wants to merge 8 commits into from

Conversation

jhamman
Copy link
Member

@jhamman jhamman commented Jul 28, 2019

In this PR, I'm experimenting with using the entrypoints package to support 3rd party backends. This does not attempt to solidify the API for what the store is, I feel like that should happen in a second PR. Here's how it would work...

In @rabernat's xmitgcm package, there is a _MDSDataStore that inherits from xarray.backends.common.AbstractDataStore. To allow reading mds datasets directly in xarray.open_dataset, xmitgcm would add the following lines to its setup.py file:

setup(
    ...
    entry_points={
        'xarray.backends': [
            'mds = xmitgcm.mds_store:_MDSDataStore',
            ...
            ]
        }
    )

Xarray would then be able to discover this backend at runtime and users could use the store directly in open_dataset calls like this:

ds = xr.open_dataset('./path/to/file.mds', engine='mds', backend_kwargs={...})

Note: I recognize that xmitgcm.open_mdsdataset has a bunch of other user options that I'm likely ignoring here but this is meant just as an illustration.

Now a list of caveats and things to consider:

  1. I have only done this for open_dataset, not for to_netcdf. We may want to consider more generic serialization method that allows for plug-able writers.
  2. open_dataset has some special handling for some readers (lock and group selection, file-like objects, etc.). We should work toward moving as much of that logic into the Store objects as possible.
  3. We should decide what to do when a 3rd party plugin conflicts with an existing backend. For example, someone could include an entrypoint with the key of netcdf4.

@pep8speaks
Copy link

pep8speaks commented Jul 28, 2019

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-07-31 21:12:25 UTC

@alexamici
Copy link
Collaborator

@jhamman & @shoyer the refactor of the big if-chain to a dictionary in the form {backend_name: backend_open} is useful independently of the using the plugin interface, would make this PR smaller and may be useful in the bigger backend refactort.

If you don't mind I'll ask @TheRed86 to open a PR for this refactor alone.

@TheRed86 TheRed86 mentioned this pull request Sep 16, 2020
3 tasks
jhamman pushed a commit that referenced this pull request Sep 24, 2020
…e: backend_open}. (#4431)

* Add docs re stable branch (#4444)

* Add docs re stable branch

* Update HOW_TO_RELEASE.md

Co-authored-by: keewis <keewis@users.noreply.github.com>

Co-authored-by: keewis <keewis@users.noreply.github.com>

* Port engine selection refactor from #3166 and add zarr

* Always add `mode="r"` to zarr and simplify logic

Co-authored-by: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>
Co-authored-by: keewis <keewis@users.noreply.github.com>
Co-authored-by: Monica Rossetti <m.rossetti@bopen.eu>
@aurghs aurghs mentioned this pull request Nov 12, 2020
3 tasks
@jhamman
Copy link
Member Author

jhamman commented Jan 12, 2021

closed by #4577

@jhamman jhamman closed this Jan 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

API Design for Xarray Backends
4 participants